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ABSTRACT 

A primary goal of university instruction is the students' demonstration of im¬ 
proved, highly developed critical thinking (CT) skills. However, how do faculty 
encourage CT and its potential concomitant increase in student workload without 
negatively impacting student perceptions of the course? In this investigation, an 
advanced biology course is evaluated after structural changes (implemented in 
2010) met with a poor student evaluation of the course and the instructor. This 
analysis first examines the steps used to transform a course to encourage CT and 
then explains how it can be assessed.To accomplish these goals, the instructor 
collaborated with an educational developer to redesign the course using a phi¬ 
losophy informed by SoTL This approach, as we see it, represents a set of prin¬ 
ciples that demand transparency in the development and application of strate¬ 
gies whose aim is to encourage student learning. However, the SoTL approach 
would be insufficient to simply promote a set of strategies without some mecha¬ 
nism for evaluating its efficacy. Therefore, we designed a "Graded Response"(GR) 
multiple-choice test to measure CT development and hence to properly evaluate 
whether the strategies embedded in our SoTL-informed course redesign have 
adequately met our goals. 
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INTRODUCTION 

The Scholarship of Teaching and Learning (SoTL) begins with moments of reflection 
on what students are learning (or not), and why. Brookfield (1997) emphasizes the role of 
reflection whereby instructors should examine their performance through a lens of what 
he termed "critical mirrors” (Brookfield, 1997, p. 19). He encourages inviting criticism 
from peers to facilitate deeper thinking about teaching and learning, a necessary part of 
course redesign: “When our peers listen to our stories and then reflect back to us what 
they see and hear in those stories, we are often presented with a version of ourselves and 
our actions that comes asasurprise” (Brookfield, 1997, p.l9).Ghaye (2011) also argued 
for reflection, what he termed “Reflected Practice,” as a key attribute in the development 
of a course aimed at encouraging deeper understanding. Reflection also plays a role in 
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Biggs’s (1996 & 1999) notion of “constructive alignment” in course design: only through 
personal reflection on the part of the instructor could the alignment of course objectives 
and assessment tasks be integrated in a manner that encourages deeper learning. Brook¬ 
field also connects reflection on the teaching and learning context with the goals for stu¬ 
dent when he states, “Accurate assessment springs from an informed understanding of 
the phenomenon being studied, and the assessment of critical thinking is no exception” 
(Brookfield, 1997, p. 19). In summary, reflection should play a key role in course redesign 
aimed at critical thinking (CT), as well as the methods for assessing the revisions. 

Although reflection isn’t always listed explicitly as a step in SoTL, it’s implied in many 
of its definitions and descriptions. Felten (2013) eloquently details five principles for 
evaluating SoTL: inquiry into student learning, grounded in context, methodologically 
sound, conducted in partnership with students, and appropriately public (p. 122). His is 
a useful list in developing SoTL projects and programs, but I argue that these “principles 
of good practice in SoTL” are also principles of good practice in course redesign aimed at 
deeper understanding. His principles coupled explicitly with reflected practice (Ghaye, 
2011) and constructive alignment methodologies (Biggs, 1996; Biggs & Tang, 2007) are 
here presented as a template for course design. Furthermore, to test the effectiveness of 
the proposed template, this essay will assess student learning in the course redesign using 
a Graded Response (GR) test, which is based on the Watson Glaser Critical Thinking 
Appraisal® (WGCTA - Watson & Glaser, 2006). 

Altogether, this framework establishes a SoTL-guided approach to course develop¬ 
ment and redesign grounded in reflection, constructive alignment, an exploration of the 
pertinent literature, and experimental analysis. This paper is divided into two parts: the 
first part will explain the rationale for applying the principles of SoTL, whereas the sec¬ 
ond part provides the study of an example of a course resulting from this redesign frame¬ 
work. This course underwent a number of changes aimed at increasing the critical think¬ 
ing in an upper-year biology course in a large Canadian university with a reputation for 
academic research excellence. The cohort under investigation represents a highly diverse 
demographic comprised of domestic and international students. 

PART 1: APPLYING FELTEN'S SOTL PRINCIPLES TO COURSE 
REDESIGN 

"Inquiry focused on student learning" 

Applying Felten’s first principle of SoTL to course redesign methodology ensures 
that the instructor starts by reflecting on the students’ learning. It encourages a specificity 
by situating that learning (or lack thereof) in the context of the course’s precise structure 
(i.e., lecture-delivery model, assignment goals and expectations, alignment with specific 
outcomes, and assessments). It is essential—in SoTL as in course redesign—to begin by 
asking what it is that we want our students to learn and perhaps even more importantly 
how we want the students to learn it. In fact, Trigwell (2013) claims that students’ ap¬ 
proach to learning—surface or deep—reflects the presence (deep) or absence (surface) 
of the instructor’s SoTL approach to teaching. 

There is ample evidence to suggest that today’s student functions in a learning en¬ 
vironment that is based on the avoidance of failure (Covington, 1984; Keeley, Shem- 
berg, Cowell, & Zinnbauer, 1995; Seifert, 2004). Hounsell, McCune, Litjens, & Hounsell 
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(2005) observed that early-year students are much more likely to exhibit a pronounced 
fear of failure, while upper-year students have a fairly sound comprehension of what is 
expected within their field. However, this recognition of what is expected may not be 
manifested in the latter students’ attitude toward a particular course, as many university 
students resist active engagement and hence remain passive in their approach to learning, 
which students perceive as carrying less risk (Haas & Kelley, 1998; Leshowitz, DiCerbo, 
& Symington 1999; Seifert, 2004). Indeed, even students who already possess many of 
the relevant CT skills may resist exercising such skills in the absence of direction from an 
“esteemed peer” (to use Mathews’ and Lowe’s term [2011]). This failure-aversive resis¬ 
tance to CT has been discussed by Keeley et al. (1995), Tsui (2002), and Mathews and 
Lowe (2011), who argue that engaging in CT requires a supportive environment. Simi¬ 
larly, Leshowitz, DiCerbo, & Symington (1999) claim that “pseudoknowing” or surface 
learning can be replaced by “reasoned knowing” only if the appropriate support mecha¬ 
nisms are in place. Starting with this concern about student learning, course redesign 
frames the course with guidance, support, and opportunities for students to successfully 
demonstrate this learning. 

"Grounded in context" 

Felten claims that good practice in SoTL is also grounded in its specific contexts. Evi¬ 
dence suggests that the most effective learning also takes place when it is context-based 
(Entwistle, 2000). This principle is critical in order to fully apply interventions that en¬ 
courage the desired learning—especially CT. For instance, performance must be mod¬ 
eled for the students in a manner that addresses both the prior iterations of the course, 
as well as the standards of the discipline. Simply demonstrating the preferred method 
without these contexts is insufficient. Actively deconstructing a sample paper with the 
class within the context of disciplinary and course expectations is another approach. In 
other words, instructors model their own approach to evaluating a paper by examining 
the questions that are proposed by the authors and then consider the author’s findings 
in order to evaluate whether the questions have been addressed in a robust manner. Key 
components would include assessing how effectively questions are addressed, examining 
competing methodologies, and deriving a summary of the differences and similarities. 
By building into a course plenty of exercises or tutorials that encourage the investigative 
process, students learn how to recognize the quality of arguments and the difference be¬ 
tween an opinion and a supported hypothesis or claim. Situating an intervention within 
the larger context of the learning goals of the course and the discipline helps scaffold the 
support at a time when the students most need to practice the relevant skill. 

"Methodologically sound" 

Another key attribute ofFelten’s (2013) principles for evaluating SoTL are the strength 
or soundness of its methodology, again a criterion that can be applied to course redesign. 
The previous steps in this course redesign framework require instructors to revisit the 
goals and contexts for student learning. Trigwell (2013) encourages us to start earlier by 
evaluating our approach to teaching. Assessments, for example, should reflect both the 
learning goals and the intentions in teaching. Trigwell describes assessment intended 
to measure content acquisition as an “Information Transfer / Teacher-Focused Model” 
(ITTF), a conception of teaching that limits students’ ability to experience deeper un- 
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derstanding (p. 98). Revisiting the approaches to teaching and the learning provides a 
sounder course redesign methodology, one that is more likely to lead to what Trigwell 
refers to as a “Conceptual Change / Student-Focused” model (CCSF). 

"Conducted in partnership with students" 

Felten (2013) describes a principle of mutually supportive roles played by both 
instructor and student in SoTL, a partnership that also works well in course redesign. 
Scott, Lisagor, and Marachi (2009) describe the changing face of American universities 
as involving a paradigm shift from a pedagogy that focused on direct instruction to one 
that construes the entire instructor-student experience in terms of collaboration. In es¬ 
sence, this philosophy points to a responsibility on the part of the instructor to provide 
suitable incentives that encourage students to participate in their own learning process. 
The literature describes many learning environments as a by-product of the contempo¬ 
rary university cohort (Abualrub, Karseth, & Stensaker, 2013; Aldridge & Fraser, 2000; 
Aldridge, Dorman & Fraser, 2004; Blair, 2013) The absence of a SoTL approach to course 
design may directly contribute to the course’s failure, as the instructor has misjudged the 
students’ learning and various contexts and leaves unexamined the entire approach to the 
classroom. This can leave students ill prepared and not properly guided through the in¬ 
vestigative and reflective behaviours that are vital for creating a CCSF environment more 
likely to support CT. Lizzio, Wilson, & Simons (2002) point out that students appear to 
develop a better and, hence, deeper understanding of course material if they have more 
time to reflect on and reinforce their learning. As a result, courses that are perceived as 
having a very high workload and affording less time for reflection are, in Lizzio et al.’s 
(2002) estimation, not as likely to promote CT. 

"Appropriately public" 

Felten’s final principle of SoTL supports the commitment of instructors to the shar¬ 
ing of their SoTL. The reasons for going public in SoTL apply to course redesign as well. 
Making available for peer review the process and details of course redesign provides 
a means for other instructors to examine, evaluate, and potentially adapt to their own 
courses. This SoTL-inspired template thus serves as a way to make course redesign pub¬ 
lic as a transparent and thus reproducible framework. A key trait of this course redesign 
template, therefore, is not only to provide a method for redesigning a course, but also to 
ensure that the entire process can be assessed through its exposure to academic scrutiny. 

PART 2: SOTL REDESIGN IN PRACTICE: AN APPLICATION 
The Redesign 

As a sabbatical replacement in 1996, deBraga taught Major Features of Vertebrate 
Evolution (BI0356), an advanced evolutionary biology course. The course had been 
delivered in a traditional manner with a focus on the anatomical diversity of vertebrate 
life throughout the earth’s history, and its variety of instructors focused on measuring 
content acquisition and, to a much lesser degree, on assessing students’ metacognitive 
abilities and “deeper understanding.” In all iterations of the course, its structure remained 
relatively static until 2010. 

In the 2010 iteration of BI0356, the laboratory components (including fossil exami¬ 
nation labs) were replaced with tutorials designed to support new assignments aimed at 
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offering students extra support for their learning. However, it failed: the absence of the 
opportunities for reflection and feedback on the part of the students during the tutorials 
proved problematic, and the changes in the course had an unanticipated and significantly 
negative impact vis-a-vis the learning centered ideal embedded in Trigwell’s CCSF ap¬ 
proach (Brookfield, 1997; Froyd & Simpson, 2010). The dramatic change in focus from 
traditional, predominantly content-driven assessment resembling the ITTF model to a 
more theoretical set of investigative assignments (i.e., assignments that tasked students 
to evaluate the merits of competing hypotheses) without a concomitant consideration 
of how the students would experience and learn (or not) these new approaches proved 
problematic. Student opinion surveys (SOS) demonstrated the negative consequences 
of the changes (see Table 2), particularly suggesting a course design lacking in Felten’s 
first principle of using the student learning as the touchstone. 

In addition, little attention had been paid to the demographic nature of the course’s 
2010 version and how much had changed since my previous experience over a decade 
earlier (1996). An entirely new generation of students had arrived with an institutional 
memory of a very specific view of the course and its structure. Students were not well 
prepared for the complexity of the tasks that these new assignments demanded, and the 
course revision hadn’t started with their learning in mind, considered the implications 
of the context of this specific student body, or offered appropriate support mechanisms, 
so an unanticipated but perhaps predictable resistance to the new course structure was 
observed (Beachford & Beachford, 2010; Kyndt, Dochy, Struyven, & Cascallar, 2011; 
Taylor, 2006). Finally, the approach to assessment was in need of review. As we began 
the course redesign, we paid particular attention to how the assessments were designed 
(or not) for the specific student learning goals of CT. 

Beginning with the student learning 

The active role of the educational developer at the start of the process and in on¬ 
going assessment of the changes in the course strengthened the redesign methodology 
and ensured proper scrutiny (Felten’s 3 rd principle). This collaboration also ensured that 
we started with—and continued to focus on—the student learning. The entire course 
syllabus was revised by establishing the learning outcomes of the course and using a back¬ 
ward design to accomplish those goals (Prideaux, 2003), an approach that meets Felten’s 
first principle. That process eventually led to a more interactive approach to delivery of 
in-class materials, including CT-focused tutorials and assignments. The resulting GRtest 
represented an entirely new form of multiple choice (MC) test, which was based on the 
inference capacity of the WGCTA. More significantly, it changed the primary mode of 
learning and assessment in the course from a surface ITTF to one which demanded a 
deeper understanding on the part of the students. 

A Focus Group Approach to "Partnership with Students" 

The implementation of a focus group or some other intervention for gathering real¬ 
time feedback from the students established what Felten might consider more of a partner¬ 
ship with the students in supporting their learning. The focus group was used as a quali¬ 
tative assessment of student feedback during the 2011 and 2012 iterations of the course. 
This feedback, coupled with student opinion survey (SOS) data, helped to frame and 
reinforce the general student perceptions of the course. The use of a focus group permits 
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a real time reflective component as well as a post-course reflective component, both of 
which are aimed at making sure the instructor understands the ever-changing student 
demographic while adhering to intended learning goals. 

Ultimately among the different focus group types, it was decided to pursue an ex¬ 
ploratory methodology (Morrison, 1997), which benefits from the informal nature of its 
design. The premise of using this methodology is the sense of “safety” that the students 
feel with the educational developer when discussing their attitude toward the course 
and the instructor. The focus group session took place during a regularly scheduled tu¬ 
torial session when neither the course instructor nor the TA was present. Students were 
assured of the anonymity of their responses and were encouraged to be honest in their 
evaluation of the course and the instructor. The questions that were put to the students 
were divided into two categories: on the one hand, lecture and tutorial content and, on 
the other, evaluation of the instructor. (See Appendix 1 for focus group questions.) The 
questionnaire served as an opportunity for the students to consider their overview of 
the course while also providing feedback to the instructor at a point that would permit 
some real-time reflection. The value was qualitative, and it was not our intent to formally 
review the responses. Hence, responses were blended, and the educational developer 
arranged a time to discuss the feedback with the instructor. The general impression was 
one of support for the instructor and notably for the helpful nature of the tutorials, but 
students on the whole were concerned that specific direction was lacking. More precisely, 
students were afraid that the question-generating nature of the lecture material and as¬ 
signment structure precluded clear parameters, especially concerning what they needed 
to know, echoing what much of the literature suggests about student fears of taking risks 
and desiring more certainty and security (Haas & Kelley, 1998; Leshowitz, et al., 1999; 
Seifert, 2004). 

The Intervention 

In the syllabus, in the first class, and throughout the course, the instructor was very 
forthcoming that the instructional paradigm for this iteration of the course was going to 
be much more demanding than previous iterations. Ghaye (2011) considers the time and 
effort that are required for this level of interaction as needing justification. As this course 
transformed from its 2010 iteration through into the final 2013 iteration, it became clear 
that modeling the appropriate student behaviors was necessary, especially in terms of re¬ 
flective practice as a foundation for CT skills. 

One tutorial in particular was specifically designed to prepare the students for an 
assignment that tasked them with evaluating the merits of competing phylogenetic hy¬ 
potheses. In this instance, the students would have to assess the degree of accuracy of the 
competing arguments or evaluate the degree of “truth” of the competing views. The idea 
of “truth” or truthfulness is an extension of the underlying philosophy of the WGCTA and 
is fundamental to the GR test, the specific assessment we developed during this course 
redesign. The most critical aspect of the tutorial design is not summative but instead a 
formative intervention. 

Students were paired up randomly or, if numbers necessitated, put into groups with 
a maximum of three participants. Students were then each assigned a different article 
that proposed competing hypotheses and were asked to identify inconsistencies or dif¬ 
ferences in both the arguments and methodologies. The course instructor or TA then 
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asked each group to discuss their paper and determine if a consensus could be reached 
regarding the proposed hypotheses. After a short period of time, groups were asked to 
deliver their criticism, which was then discussed within the class. The value to this ap¬ 
proach is that students were free to present their opinions in a risk-free, low-stakes envi¬ 
ronment. All student opinions, which were supported with evidence, were assumed to 
have value as they were part of the reflective practice that had been emphasized at the 
start of the tutorial. The aim of the exercise was to help in the development and honing 
of formal arguments that could be used to better support a particular claim. Although 
not all groups were able to present their results during the one-hour tutorial, there were 
opportunities to continue discussions of the activity (and the overall course) through a 
discussion board. Students were encouraged to participate but were not given credit for 
doing so. Instead, either the course instructor or TA regularly addressed the concerns on 
the discussion board that would benefit from looking to the course objectives and em¬ 
phasized the importance of active participation in these discussions for their assignments 
and other summative assessments that were to follow. 

Assessing the "soundness" of the redesign methodology 

To assess the interventions implemented in the course redesign between 2010 and 
the fully supported courses of 2012 and 2013, it was necessary to quantitatively measure 
the impact of the modified tutorials and the GR test. This process focused on the efficacy 
of the WGCTA as the template for the course’s new assessment (the GR test), as well as 
data collected from the various course iterations (pre-intervention vs. post-intervention) 
regarding any impact on CT. Additionally, this process involved comparing SOSs in the 
pre- and post-intervention iterations of the course to identify the challenges resulting 
from such a significant structural change in the course, notably transitioning from mea¬ 
suring content acquisition to an investigative design. 

The WGCTA (Watson & Glaser, 2006) was designed to test CT skills among profes¬ 
sionals and students alike. The version of the test applied in this investigation represents 
the WGCTA “Short Form” which was first published in 1994 and has been revised as re¬ 
cently as 2005. The test is designed to measure five specific skills: inference, recognition 
of assumptions, deduction, interpretation, and evaluation of arguments. The WGCTA has 
been used quite extensively in the professional world and notably in professional university 
programs such as nursing, where it has repeatedly demonstrated its reliability as a measure 
of CT (+/- 1.96 at 95% confidence limits; Magnussen, Ishida, & Itano, 2000). Of the five 
WGCTA skills, the inference testing demonstrated the greatest application to our GR test 
design, based on how the logic inherent in this type of question 1 suits the aim of the GR 
test and serves as a means to encourage CT. Notably, the form of this question requires 
that a student can determine the degree of truthfulness for each of the statements. In the 
example below, we compare a question from a traditional pre-intervention MC test (la) 
to the modified GRtest questions (lb) from the course redesign. Note that the bolded 
response in la refers to the correct response. In figure lb a full mark is awarded only if 
the correct order from best to worst (i.e., 1= best, 4 = worst) is assigned by the student. 
If the best and the worst responses are identified correctly but the remaining order is in¬ 
correct, only one-half mark is awarded. Alternatively, if only the best choice is selected 
and the rest are incorrect, one-quarter mark is awarded. A grade of zero is applied only if 
all options are incorrectly ordered. 
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la. (WINTER 2010) In the view of the fossorial origin of snakes,... 

A. mosasaurs are believed to be the closest sister-group because of their long tails. 

B. mosasaurs are believed to be the closest sister-group because of their streptostylic 
skulls. 

C. amphisbaenians are believed to be the closest sister-group because of their 
burrowing habits. 

D. their delicate skulls are interpreted as being an adaptation for burrowing. 

E. legs are believed to have served a burrowing role in early members. 

lb. (WINTER2012-13) Which statement best reflects the current understanding? (Rank 

the statements.) 

4-A. Snakes have separate, independently moveable lower jaw elements. 

2- B. Snakes possess a ligamentously attached jaw symphysis permitting extensive flexi¬ 

bility during feeding. 

1-C. Snakes possess a lower jaw symphysis that is loosely connected by a ligament per¬ 
mitting autonomous movement of each half of the lower jaw thereby facilitating 
the ingestion of large prey. 

3- D. Snakes have a flexible lower jaw designed to permit the ingestion of large prey. 

In question lb, the best response (l-C) corresponds to the “TRUE” statement whereas 
the worst response (4-A) would correspond to the “FALSE” statement as expressed in 
the WGCTA test. Therefore, students were no longer required to simply identify correct 
responses but were, in essence, being tasked with evaluating why some responses are bet¬ 
ter than others. This requirement forces the student to justify a rationale with respect to 
how they rank the statements and hence it is hypothesized that this approach promotes 
the development of CT (Thomas, 2011). In an effort to validate the application of the 
GR test 2 —and thus the effectiveness of the course redesign—a statistical analysis helped 
investigate the relative merit of the test. 

We asked students who were registered in the course during the 2012 and 2013 (n= 
55) iterations to voluntarily participate in the WGCTA assessment. The two cohorts were 
combined (n=34; 3 were later excluded for failing to complete all course requirements) 
and compared to students from the 2011 iteration of the course (n=22) to test whether 
there were any differences between the two populations. 

The test was administered during a regularly scheduled tutorial session, which had 
been assigned as an open date (neither the course instructor nor the course TA were in 
attendance). A member of the academic skills centre who was not directly affiliated with 
the course delivered the test. Student performance on this test was not shared with the 
course instructor. At the end of the course and after all grades had been submitted, the 
data was analyzed using SPSS and subjected to a regression analysis designed to determine 
if there was any correlation between the test scores on the GR and WGCTA tests. Data 
from 2012 and 2013 were pooled and students’ final scores on the GR and assignments 
were compared with the WGCTA performance scores. In order to verify the capacity of 
the GR test to serve as a measure of CT performance, the WGCTA was examined for 
correlation with the GR test using a paired samples T-test. A probability plot was con¬ 
structed to test normality of the scores for each variable. Given the small “n” for students 
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who participated in the WGCTAin each of the years (ll in 2012,23 in 2013) it was nec¬ 
essary to determine if data from the test groups could be pooled. Application ofLevene’s 
test to examine whether there was any statistically significant variance between the two 
test group populations was undertaken and followed by an independent sample T-test. 
(See results section for p-values and other relevant data.) 

In addition to examining the student performance of CT, we also evaluated the im¬ 
pact of the course changes on perceptions of the course instructor through the use of in¬ 
dependently 3 administered SOS using a Likert scale questionnaire scored out of a total 
of seven. For the specific question about the instructor (“The instructor created a course 
atmosphere that was conducive to my learning.”), an average score ranges from 5.6 to 5.8 
with a strong score reading from 5.9 to 6.0. A score above 6.0 is regarded as exceptional; 
scores under 5 are considered weak. There is also an opportunity for students to offer 
personal commentary. 

RESULTS 

Table 1 presents the statistical significance of the course assessment tools as they 
relate to their capacity to measure CT by examining the relative performance of a group 
of students (31) enrolled in the course whose 2012 and 2013 data was pooled. Note 
that the grouping of the 2012 and 2013 WGCTA data is possible as there was no vari¬ 
ance in between the two iterations of the WGCTA test populations (p-value of 0.158 & 
0.477). In addition a paired samples T-test examining the correlation between the 
WGCTA and GR tests has a p-value of < 0.001 with a moderate positive correlation of 
0.388, suggesting that the GRtest is a viable alternative measure of CT in the absence of 
the WGCTA. 

There also appears to be a trend in student performance according to test scores. For 
example, students in the test group exhibit a decline in their MC test scores from 65% in 
the control group down to 56% for the test group (p-value 0.048) while simultaneously 
displaying an overall increase in the total score for the final exam, moving from 65% to 
69% (p-value 0.118) respectively when GR and MC scores are combined. Although this 
upward trend is not statistically significant, it might suggest an attempt on the part of 
students to dedicate more time in their preparation for GR test questions, which could 
be interpreted as a commitment to explore deeper understanding. At the very least, the 


Table 1. Justification for the pooling of 2012 and 2013 WGCTA student performance data as given by the absence of 
any statistically significant difference in student performance (p-values of 0.158 & 0.477). Listing of mean scores (as 
%) and standard deviations (SD) along with confidence intervals and relevant p-values for assignments and summa- 
tive assessment tasks including exams using standard MC questions and those that combined MC and GR questions. 



CONTROL GROUP 

TEST GROUP 

95% CONFIDENCE 


ATTRIBUTE 

MEAN 

±SD 

MEAN 

±SD 

INTERVALS 

P-VALUE 

Assignment # 1 

78 

±7.4 

74 

±13.7 

-1.7-10.1 

0.158* 

Assignment # 2 

72 

±9.1 

69 

±14.6 

-4.3-9.1 

0.477* 

Exam MC vs. MC 

65 

±10.6 

56 

±17.2 

0.9-16.7 

0.048 

Exam MC vs. GR+MC 

65 

±10.6 

69 

±9.0 

-9.9-1.1 

0.118 


* (asterisk) refers to values reflecting no variance between means as determined by the independent T-test for equality of means. 
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Table 3. Comparison of course structure, pre-intervention vs. post intervention models 


PRE-INTERVENTION (2010) 

POST INTERVENTION (2011-13) 

Syllabus informative but insufficiently descriptive. 

Syllabus fully aligned, linking course objectives to 
assessment tasks and course materials. 

Standard multiple choice testing combined with 
short answer and essay questions. 

Graded response (GR) testing combined with short 
answer and essay questions. 

Assignment essays include outline and general 
rubric. Multiple drafts not available. 

Assignment essays include outline, detailed rubric, 
and use of multiple drafts aimed at encouraging 
student self-reflection. 

Tutorials aimed at content specific support. 

Tutorials redesigned to encourage student feedback 
through modeling of CT behaviours. 

Student opinion not sought out formally, other than 
through the university’s SoS. 

Student opinion sought formally through embedded 
focus group with an emphasis on reflection, as well 
as SoS. 

Instruction tending to focus on “what” or content 
acquisition. 

Instruction focusing on “why” or justification of 
rationale. 


findings suggest that students may be spending somewhat less time memorizing content- 
specific statements and committing more time to exploring concepts that require greater 
reflection. An additional finding as provided by the SOS scores was the fundamental shift 
in the response of the students from generally negative to significantly positive. Table 2 
above compares the overall rating of the instructor from the pre-intervention in 2010 to 
the 2012 iteration. Included in the table are the respective SOS scores and the percent of 
respondent rate for these surveys. 

Additionally prior to the redesign, students had been quite critical of the course 
expectations, offering frequent statements such as “too much information . . . delivered 
at a stifling pace.” By 2012, the comments had changed dramatically as evidenced by this 
comment: “Professor is very enthusiastic and simplifies a complicated concept. Focuses 
on the key concepts for testing information. Expects students to understand and not 
just memorize important facts. I really enjoyed his class.” Given the aim of the course re¬ 
design, a comment of this type represents an opportunity to validate, albeit in a qualita¬ 
tive manner, student willingness to recognize their responsibility in their own learning. 
In our opinion, this view suggests the student’s acceptance that it is insufficient to col¬ 
lect facts and that there is greater value in determining how or when to apply these facts. 

CONCLUSION 

Collaborating with the educational developer on a course design through the lens 
of SoTL principles of good practice allowed us to shift toward a learning centered envi¬ 
ronment that provided the students with opportunities to develop their CT skills. With 
assessment tools that can effectively validate the course’s CT goals through a standard¬ 
ized CT test like the WGCTA or a discipline-specific tool like the GR test, we can not 
only support the development of CT but also verify its presence. Effectively, we have 
developed a step-wise process (Table 3) that can ensure appropriate alignment between 
course outcomes and course delivery. 
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We acknowledge that there may be faculty resistance to implementing the neces¬ 
sary changes, but the benefits significantly outweigh the costs (Gardiner, 1998). Where 
resources are available, it is advantageous for university faculties to place a high priority 
on the development of pedagogically sound practices that will benefit both the student 
and the instructor. Much of the SoTL literature (Bernstein 2008; Felten, 2013; Trigwell, 
2013) emphasizes the need to share in teaching and learning projects (specifically SoTL, 
by also by extrapolation, course redesign), so that there is a greater sense of objectivity 
and pedagogical value when evaluating the merits of any given instructional strategy. A 
collaborative effort is, therefore, ideal to use in course redesign processes, especially those 
that apply the principles of good practice in SoTL. 

Michael deBraga is a lecturer supporting mathematics and science in the Robert Gillespie Academic 
Skills Center and teaches in the Biology Department at University of Toronto Mississauga (CANADA). 

Cieo Boyd is a senior lecturer and educational developer in the Robert Gillespie Academic Skills 
Center at University of Toronto Mississauga (CANADA). 

Shahad Abdulnour is an academic strategist in the Robert Gillespie Academic Skills Center and 
teaches in the departments of Historical Studies and Biology at University of Toronto Mississauga 
(CANADA). 

NOTES 

1. Here is an example from the WGCTA Short Form test booklet (Watson & Glaser, 2006, pg.3). 
"Assign to each statement a designation as to whether it is TRUE, PARTLY TRUE, INDETER¬ 
MINATE, PARTLY FALSE, or FALSE: Two hundred students in their early teens voluntarily at¬ 
tended a recent weekend student conference in a Midwestern city. At this conference, the 
topic of race relations and means of achieving lasting world peace were discussed, because 
these were the problems the students selected as being most vital in today's world. 

1. As a group, the students who attended this conference showed a keener interest in the 
broad social problems than do most other students in their early teens. 

2. The majority of the students had not previously discussed the conference topics in 
their schools. 

3. The students came from all sections of the country. 

4. The students discussed mainly labor relations problems. 

5. Some teenage students felt it worthwhile to discuss problems of race relations and 
ways of achieving world peace." 

2. The WGCTA has been shown to effectively measure CT skills. A regression analysis on a 
sample of self-selected students was undertaken to evaluate whether there was a corre¬ 
lation between the performance of the students on the GR test and course assignments 
and the WGCTA (see results section). 

3. Student opinion surveys (SOS) are administered through the university for every course. 
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APPENDIX 1 

Lecture and tutorial content 

l) How prepared were you prior to the start of this course? For example, how con¬ 
fident were you about meeting the course expectations? 
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2) Now that you have passed the halfway point of the course, with respect to your 
performance, how confident are you that you will successfully meet the course 
expectations? 

3) Do you feel that the course pre-requisites have sufficiently prepared you for this 
course? 

4) Are course tutorials helpful in preparing you for your assignments? What do 
you find most helpful? What would you like to see added or deleted? 

5) What did you find most challenging about your most recent assignment? 

6) Regarding the visit from the science librarian, have you experienced librarian 
visits before? 

7) How did this visit differ from others you have had? 

8) Was the visit by the librarian helpful in presenting you with an overview of the 
available resources? 

Instructor assessment 

9) Does the course syllabus adequately lay out the course requirements and the 
necessary resources? For example, is the course content on Blackboard laid out 
in an easy to access manner (i.e., course information folder, course lecture folder, 
etc.)? 

10) Do you find the narrative style of your instructor enjoyable? Why or why not? 

11) Did the ROM (Royal Ontario Museum) visit provide you with an opportunity 
to interact with your peers and with your instructor in a less formal manner? 
Was this opportunity beneficial? 

12) Do you appreciate the interactive nature of how the course material is delivered? 
In other words, do you find that concepts are presented without necessarily hav¬ 
ing a resolution? 
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